I am looking for one of my clients into how costs can be reduced, or quality increased, by increasing the extent to which testing is automated.

As a first step, I am trying to develop a comprehensive list of test automation "best practices", grouped roughly by life-cycle (or iteration) stage. I'm trying to find practices which are broadly independent of specific methods and technologies, although obviously tool support may vary depending on the chosen technology.

The following is my first draft of such a list. The tools mentioned are simply examples - there are several tools available in each category, both full cost and open source.

I'd welcome suggestions from my readers if you think there are any omissions (or if you substantially disagree with anything I've included).

Thanks

Andrew

1. Suite of automated unit and system tests, which can be run on demand by the developers and before integration.
2. Test-driven development process, in which unit tests are developed before the code they test.
3. Unit and system tests built into the product suite, using one of the xUnit family of tools.
4. Daily or continuous integration process, which includes execution of the test suite.
5. Use of virtual machines or other "cloning" technologies to create easily reproducible development and test environments.
6. Code coverage measured using a tool such as NCover, and demonstrated to be 100% (or as near as practically possible) for all code.
7. Code coverage validated using mutation testing (using a tool such as Jester) to confirm that every line can affect the outcome of at least one test.
8. Feedback process to enhance test suite for bugs discovered in later test stages or live use.

1. Automated end-to-end tests, run for every release.
2. Functional coverage measurement, possibly combined with traceability mechanism between requirements and tests.
3. Ability to easily create and reset an environment which functionally matches production.
4. Automated test management and reporting, providing control/confirmation of tests run, pass/fail results, errors and diagnostics and resolution tracking.
5. Repeatable and automated performance and/or stress tests
6. Ability to inject/extract test messages and files into and out of interfaces and to track them through any integration scheme, ideally in production as well as test environments.
7. Repeatable, pre-prepared set of test messages/files which test all interfaces, including exception handling.
8. Feedback process to enhance test suite for bugs discovered in later test stages or live use.

The nature of user acceptance test is that most functional testing will be manual, since this will provide maximum independent assurance to the users. The following focuses on where automation can best support the process:

1. Ability to easily create and reset a UAT environment which functionally matches production.
2. Automation for set-up, background and repetitive tests.
3. Automated test management and reporting, providing confirmation of tests run, pass/fail results, errors and diagnostics and resolution tracking.
4. Ability to inject/extract test messages and files into and out of interfaces and to track them through any integration scheme, ideally in production as well as test environments.
5. Feedback process to enhance test suite for bugs discovered in live use.

1. Automated and non-destructive end-to-end and integration tests, which can be run on demand following any production change, or as an initial diagnostic action.
2. Ability to inject/extract test messages and files into and out of interfaces and to track them through any integration scheme.
3. Regular automated execution of key tests to verify system performance.
4. Automated test management and reporting, providing confirmation of tests run, pass/fail results, errors and diagnostics and resolution tracking.